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Executive Summary 



The District of Columbia School Choice Incentive Act of 2003, passed by the Congress in 
January 2004, established the first federally funded, private school voucher program in the United States. 
As part of this legislation, the Congress mandated a rigorous evaluation of the impacts of the Program, 
now called the DC Opportunity Scholarship Program (OSP). This report presents findings from the 
evaluation on the impacts 2 years after families who applied were given the option to move from a public 
school to a participating private school of their choice. 



The evaluation is based on a randomized controlled trial design that compares the outcomes 
of eligible applicants randomly assigned to receive (treatment group) or not receive (control group) a 
scholarship through a series of lotteries. The main findings of the evaluation so far include: 

• After 2 years, there was no statistically significant difference in test scores in 
general between students who were offered an OSP scholarship and students who 
were not offered a scholarship. Overall, those in the treatment and control groups 
were performing at comparable levels in mathematics and reading (table 3). 

• The Program had a positive impact on overall parent satisfaction and parent 
perceptions of school safety, but not on students’ reports of satisfaction and safety 

(tables 4 and 5). Parents were more satisfied with their child’s school and viewed the 
school as less dangerous if the child was offered a scholarship. Students had a different 
view of their schools than did their parents. Reports of dangerous incidents in school 
were comparable for students in the treatment and control groups. Overall, student 
satisfaction was unaffected by the Program. 

• This same pattern of findings holds when the analysis is conducted to determine 
the impact of using a scholarship rather than being offered a scholarship. Twenty- 
six percent of students who were randomly assigned by lottery to receive a scholarship 
chose not to use it in either the first or second year. We use a common statistical 
technique to take those “never users” into account; it assumes that the students had 
zero impact from the OSP, but it does not change the statistical significance of the 
original impact estimates. Therefore, the positive impacts on parent views of school 
safety and satisfaction all increase in size, and there remains no impact on academic 
achievement and no overall impact on students’ perceptions of school safety or 
satisfaction from using an OSP scholarship. 

• There were some impacts on subgroups of students, but adjustments for multiple 
comparisons indicate that these findings may be due to chance. There were no 
statistically significant impacts on the test scores of the high-priority subgroup of 
students who had previously attended schools designated as in need of improvement 
(SINI). However, being offered or using a scholarship may have improved reading test 
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scores among three subgroups of students: those who had not attended a SINI school 
when they applied to the OSP, those who had relatively higher pre-Program academic 
performance, and those who applied in the first year of Program implementation. The 
Program may also have had a positive impact on school satisfaction for students who 
had previously attended SINI schools. However, these findings were no longer 
statistically significant when subjected to a reliability test to adjust for the multiple 
comparisons of treatment and control group students across 10 subgroups; the results 
may be “false discoveries” and should therefore be interpreted and used with caution. 

• The second year impacts are generally consistent with those from the first year . 1 

The main difference is that after 1 year, the non-SINI and higher performing groups of 
students appeared to experience statistically significant positive impacts on math 
achievement, while in the second year the impacts were on reading achievement. 
Adjustments for multiple comparisons suggest that both sets of results may be false 
discoveries. 



DC Opportunity Scholarship Program 

The purpose of the new scholarship program was to provide low-income residents, 
particularly those whose children attend schools in need of improvement or corrective action under the 
Elementary and Secondary Education Act, with “expanded opportunities to attend higher performing 
schools in the District of Columbia” (Sec. 303). The scholarship, worth up to $7,500, could be used to 
cover the costs of tuition, school fees, and transportation to a participating private school. The statute also 
prescribed how scholarships would be awarded: (1) in a given year, if there are more eligible applicants 
than available scholarships or open slots in private schools, scholarships are to be awarded by random 
selection (e.g., by lottery), and (2) priority for scholarships is given first to students attending SINI public 
schools and then to families that lack the resources to take advantage of school choice options. 

The Program is operated by the Washington Scholarship Fund (WSF). To date, there have 
been four rounds of applications to the OSP (table 1). Applicants in spring 2004 (cohort 1) and spring 
2005 (cohort 2) represent the majority of Program applicants; the evaluation sample was drawn from 
these two groups. 2 There were a smaller number of applicants in spring 2006 (cohort 3) and spring 2007 
(cohort 4) who were recruited and enrolled by WSF in order to keep the Program operating at capacity — 
approximately 2,000 students — each year. 



1 See Wolf, Gutmann, Puma, Rizzo, Eissa, and Silverberg 2007. 

2 Descriptive reports on each of the first 2 years of implementation and cohorts of students have been previously prepared and 
released (Wolf, Gutmann, Eissa, Puma, and Silverberg 2005; Wolf, Gutmann, Puma, and Silverberg 2006) and are available on 
the Institute of Education Sciences’ website at http://ies.ed.gov/ncee. 
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Table 1. OSP Applicants by Program Status, Cohorts 1 Through 4, Years 2004-2007 





Cohort 1 Cohort 2 

(Spring 2004) (Spring 2005) 


Total 

Cohort 1 and 
Cohort 2 


Cohort 3 
(Spring 2006) 
and Cohort 4 
(Spring 2007) 


Total, All 
Cohorts 


Applicants 


2,692 


3,126 


5,818 


1,308 


7,126 


Eligible applicants 


1,848 


2,199 


4,047 


846 


4,893 


Scholarship awardees 


1,366 


1,088 


2,454 


846 


3,300 


Scholarship users in initial year of receipt 


1,027 


797 


1,824 


712 


2,536 


Scholarship users fall 2005 


919 


797 


1,716 


NA 


1,716 


Scholarship users fall 2006 


788 


684 


1,472 


333 


1,805 


Scholarship users fall 2007 


678 


581 


1,259 


671 


1,930 



NOTES: Because most participating private schools closed their enrollments by mid-spring, applicants generally had their 

eligibility determined based on income and residency, and the lotteries were held prior to the administration of 
baseline tests. Therefore, baseline testing was not a condition of eligibility for most applicants. The exception was 
applicants entering the highly oversubscribed grades 6-12 in cohort 2. Those who did not participate in baseline 
testing were deemed ineligible for the lottery and were not included in the eligible applicant figure presented above, 
though they were counted in the applicant total. In other words, the cohort 2 applicants in grades 6-12 had to satisfy 
income, residency, and baseline testing requirements before they were designated eligible applicants and entered in 
the lottery. 

The initial year of scholarship receipt was fall 2004 for cohort 1, fall 2005 for cohort 2, fall 2006 for cohort 3, and 
fall 2007 for cohort 4. 

SOURCES: OSP applications and WSF’s enrollment and payment files. 



Mandated Evaluation of the OSP 

In addition to establishing the OSP, Congress mandated an independent evaluation of it be 
conducted, with annual reports on the progress of the study. The legislation indicated the evaluation 
should analyze the effects of the Program on various academic and non-academic outcomes of concern to 
policymakers and use . . the strongest possible research design for determining the effectiveness” of the 
Program. The current evaluation was developed to be responsive to these requirements. In particular, the 
foundation of the evaluation is a randomized controlled trial (RCT) that compares outcomes of eligible 
applicants (students and their parents) randomly assigned to receive or not receive a scholarship. This 
decision was based on the mandate to use rigorous evaluation methods, the expectation that there would 
be more applicants than funds and private school spaces available, and the statute’s requirement that 
random selection be the vehicle for determining who receives a scholarship. An RCT design is widely 
viewed as the best method for identifying the independent effect of programs on subsequent outcomes 
(e.g., Boruch, de Moya, and Snyder 2002, p. 74). Random assignment has been used by researchers 
conducting impact evaluations of other scholarship programs in Charlotte, NC; New York City; Dayton, 
OH; and Washington, DC (Greene 2001; Howell et al. 2002; Mayer et al. 2002). 
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The recruitment, application, and lottery process conducted by WSF with guidance from the 
evaluation team created the foundation for the evaluation’s randomized trial and determined the group of 
students for whom impacts of the Program are analyzed in this report. Because the goal of the evaluation 
was to assess both the short-term and longer term impacts of the Program, it was necessary to focus the 
study on early applicants to the Program (cohorts 1 and 2) whose outcomes could be tracked over at least 
3 years during the evaluation period. During the first 2 years of recruitment, WSF received applications 
from 5,818 students. Of these, approximately 70 percent (4,047 of 5,818) were eligible to enter the 
Program (table 1). Of the total pool of eligible applicants, 2,308 students who were rising kindergarteners 
or from public schools entered lotteries (492 in cohort 1; 1,816 in cohort 2), resulting in 1,387 students 
assigned to the treatment condition and 921 assigned to the control condition. These students constitute 
the evaluation’s impact analysis sample and represent three-quarters of all students in cohorts 1 and 2 who 
were not already attending a private school when they applied to the OSP. 

Data are collected from the impact sample each year, starting with the spring in which 
students applied to the OSP (baseline) and each spring thereafter. These data include assessments of 
student achievement in reading and mathematics using the Stanford Achievement Test version 9 (SAT- 
9), 3 surveys of parents, and surveys of students in grade 4 and above — all administered by the evaluation 
team in central DC locations on Saturdays or weekday evenings because neither the public nor private 
schools would allow data collection on their campuses during the school day. In addition, the evaluation 
surveys all DC public and private schools each spring in order to address the statute’s interest in 
understanding how the schools are responding to the OSP. 



Participation in the OSP 

In interpreting the impacts of the OSP, it is useful to examine the characteristics of the 
private schools that participate in the Program and the extent to which students offered scholarships (the 
treatment group) moved into and out of them during the first 2 years. 

School Participation 

The private schools participating in the OSP represent the choice set available to parents 
whose children received scholarships. That group of schools had mostly stabilized by the 2005-06 school 
year. The schools that offered the most slots to OSP students, and in which OSP students and the impact 

3 Stanford Abbreviated Achievement Test (Form S), Ninth Edition. San Antonio, TX: Harcourt Educational Measurement, 
Elarcourt Assessment, Inc., 1997. 
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sample’s treatment group were clustered, have characteristics that differed somewhat from the average 
participating OSP school. Only 11.2 percent of treatment group students were attending a school that 
charged tuition above the statutory cap of $7,500 during their second year in the Program (table 2) even 
though 39 percent and 38 percent of participating schools charged tuitions above that cap in 2005-06 and 
2006-07, respectively. 4 Although 55 percent of all participating schools were faith-based (35 percent were 
part of the Catholic Archdiocese of Washington), nearly 80 percent of the treatment group attended a 
faith-based school, with more than half of them (53 percent) attending the 23 participating Catholic 
parochial schools. The average OSP student in the treatment group attended a school with 196 
students — somewhat smaller than the average of 236 (2005-06) and 242 (2006-07) students across the set 
of all participating OSP schools. 

Table 2. Features of Participating Private Schools Attended by the Treatment Group in Year 2 



Characteristic 


Weighted 

Mean 


Highest 


Lowest 


Valid A 


Schools charging over $7,500 tuition 
(percent of OSP students attending) 


11.2% 


NA 


NA 


51 


Archdiocesan Catholic schools 


52.7% 


NA 


NA 


51 


Other faith-based schools 


23.9% 


NA 


NA 


51 


Tuition 


$5,928 


$29,902 


$3,500 


51 


Enrollment 


196.4 


1,056 


20 


50 


Student N 


841 









NOTES: “Valid TV” refers to the number of schools for which infonnation on a particular characteristic was available. When a 

tuition range was provided, the mid-point of the range was used. The weighted mean was generated by associating 
each student with the characteristics of the school he/she was attending and then computing the average of these 
student-level characteristics. 

SOURCE: OSP School Directory information, 2004-05, 2005-06, and 2006-07, WSF. 



While the characteristics of the participating private schools are important considerations for 
parents, in many respects it is how the schools differ from the public school options available to them that 
matters most. In the second year after applying to the OSP, students in the treatment and control groups 
did not differ significantly regarding the proportion attending schools that offered computer labs (93 and 
92 percent), libraries (83 and 87 percent), gyms (70 and 66 percent), and art programs (90 and 86 
percent). Differences in school characteristics between the treatment and control groups 2 years after they 
applied to the OSP that were statistically significant at the .01 level included: 



4 The average tuition charged to these treatment group students who used their scholarships was $5,928 but varied between 
$3,500 and $29,902. The WSF reported that families in their second year of the Program were required to pay at least some 
money out-of-pocket for tuition in 164 cases where the tuition charged by the school exceeded the $7,500 cap. 
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• Students in the treatment group were more likely to attend schools that offered a music 
program (92 percent), an after-school program (97 percent), and special programs for 
advanced learners (45 percent) compared to students in the control group (84 percent, 
94 percent, and 33 percent for each type of program, respectively). 

• Students in the treatment group were less likely to attend a school that offered 
counselors (74 percent), tutors (63 percent), programs for non-English speakers (19 
percent), and programs for students with learning problems (55 percent) than were 
students in the control group (89 percent, 73 percent, 50 percent, and 79 percent, 
respectively, for each offering). 

Student Participation 



As has been true in similar programs, not all students offered an OSP scholarship actually 
used it to enroll in a private school. For students assigned to the treatment group, during the first 2 years 
of the Program (figure 1): 

• 26 percent (366 out of 1,387) of those offered an OSP scholarship never used it; 

• 20 percent (271) used their scholarship during some but not all of the first 2 years after 
the award; and 

• The remaining 54 percent (750 students) used their scholarship consistently for the 
entire 2 years after the lottery. 

The reasons for not using the scholarship varied. The most common reasons cited by parents 
whose students declined the scholarship and completed surveys were (figure 2): 



• Lack of available space in the private school they wanted their child to attend (29 
percent of these parents); 

• Participating schools did not offer services for their child’s learning or physical 
disability or other special needs (17 percent of these parents); and 

• Child was accepted into a public charter school (16 percent of these parents). 
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Figure 1. Proportions of Treatment Group Students Who Experienced 
Various Categories of Usage in First 2 Years 




NOTES: Data are not weighted. Valid N = 1,387. Students were identified as scholarship users 

based upon information from WSF’s payment files. Because some schools use a range of 
tuitions and some students had alternative sources of funding, students were classified as 
full users if WSF made payments on their behalf that equaled at least 80 percent of the 
school’s annual tuition. Otherwise, students were identified as partial users (1 percent to 79 
percent of tuition paid) or non-users (no payments). 

SOURCES: OSP applications and WSF’s payment files. 



Students who never used the OSP scholarship offered to them, or who did not use the 
scholarship consistently, could have found their way into other (non-OSP-participating) private schools, 
public charter schools, or traditional DC public schools. The same alternatives were available to students 
who applied to the OSP but were never offered a scholarship (the impact sample’s control group). Both 
the treatment and control groups moved between public (both traditional and charter) and private schools 
or between SINI and non-SINI schools. As a result, over the 2 years after they applied to the OSP: 



• Among the treatment group, 4 percent remained in the same school they were in when 
they applied to the Program; 71 percent switched schools once; and 25 percent 
switched schools twice. 

• Among the control group, 22 percent remained in the same school they were in when 
they applied to the Program; 57 percent switched schools once; and 21 percent 
switched schools twice. 
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Figure 2. Most Common Reasons Given by Parents for Declining to Use 
the OSP Scholarship in Year 2 




NOTES: Responses are unweighted. Respondents were able to select multiple responses, which 

generated a total of 180 responses provided by 153 parents. This equates to an average 
of 1 .2 responses per parent. Responses that were not selected are unreported. 

SOURCE: Impact Evaluation Parent Surveys. 



Impact of the Program After 2 Years: Key Outcomes 

The statute that authorized the OSP mandated that the Program be evaluated with regard to 
its impact on student test scores and school safety, as well as the “success” of the Program, which, in the 
design of this study, includes satisfaction with school choices. The impacts of the Program on these 
outcomes are presented in two ways: (1) the impact of the offer of an OSP scholarship, derived straight 
from comparing outcomes of the treatment and control groups, and (2) the impact of using an OSP 
scholarship, calculated from the unbiased treatment-control group comparison, but statistically netting out 
students who declined to use their scholarships. 3 The main focus of this study was on the overall group of 



5 This analysis uses straightforward statistical adjustments to account not only for the approximately 25 percent of impact 
sample respondents who received the offer of a scholarship but declined to use it (the “decliners”), but also the estimated 2.3 
percent of the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP 
scholarship, ended up in a participating private school (we call this “program-enabled crossover”). These adjustments increase 
the size of the scholarship offer effect estimates, but cannot make a statistically insignificant result significant. 
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students, with a secondary interest in students who applied from SINI schools, followed by other 
subgroups of students (e.g., defined by their academic performance at application, their gender, or their 
grade level). 



A previous report released in spring 2007 indicated that 1 year after application there were 
no statistically significant impacts on overall academic achievement or on student perceptions of school 
safety or satisfaction (Wolf et al. 2007). Parents were more satisfied if their child was in the Program and 
viewed their child’s school as less dangerous. Among the secondary analyses of subgroups, there were 
impacts on math for students who applied from non-SINI schools and for those with relatively higher pre- 
program test scores. Statistical adjustments for multiple comparisons suggested there is a possibility that 
the subgroup achievement impacts in year 1 were chance discoveries. 

The analyses in this report were conducted using data collected on students 2 years after they 
applied to the OSP. 



Impacts on Students and Parents Overall 

• Across the full sample, there were no statistically significant impacts on reading 
achievement (effect size (ES) = ,09) 6 or math achievement (ES = .01) from the offer of 
a scholarship (table 3) nor from the use of a scholarship. 7 

• Parents of students offered a scholarship were less likely to report serious concerns 
about school danger (ES = -.27) compared to parents of students not offered a 
scholarship (table 4); the same was true for parents of students who chose to use their 
scholarships (ES = -.34). 

• On the other hand, students who were offered a scholarship reported similar levels of 
dangerous activities at school compared to those in the control group (ES = -.01; 
table 4); there was also no impact on student reports of school safety from using a 
scholarship (ES = -.01). 

• The Program produced a positive impact on parent satisfaction with their child’s 
school, for example regarding the likelihood of grading the school an “A” or “B,” both 
for the impact of a scholarship offer (ES = .26; table 5) and the impact of scholarship 
use (ES = .33). 



An effect size (ES) is a standardized measure of the relative size of a program impact. In this report, effect sizes are expressed 
as a proportion of a standard deviation of the distribution of values observed for the study control group. One full standard 
deviation above and below the average value for a variable such as outcome test scores contains 64 percent of the observations 
in the distribution. Two full standard deviations above and below the average contain 95 percent of the observations. 

7 The magnitudes of these estimated achievement effects are below the threshold of . 1 1 standard deviations, estimated by the 
power analysis to be the study’s Minimum Detectable Effect size. 



xiii 




Overall, there were no impacts of the OSP from being offered (ES = .05 to .13; table 5) 
or using a scholarship on students’ satisfaction with his or her school. 



Table 3. Year 2 Impact Estimates of the Offer of a Scholarship on the Full Sample: Academic 
Achievement (Intent to Treat or ITT) 



Student Achievement 


Treatment 
Group Mean 


Control Group 
Mean 


Difference 

(Estimated 

Impact) 


Effect Size 


/7-value 


Reading 


621.30 


618.12 


3.17 


.09 


.09 


Math 


614.09 


613.85 


.23 


.01 


.89 



NOTES: Means are regression-adjusted using a consistent set of baseline covariates. Impacts are displayed in tenns of scale 

scores. Effect sizes are displayed in terms of standard deviations of the study control group distribution. Valid N for 
reading = 1,580; math = 1,585. Separate reading and math sample weights were used. 



Table 4. Year 2 Impact Estimates of the Offer of a Scholarship on the Full Sample: Parent and 
Student Reports of School Danger (ITT) 



School Danger 


Treatment 
Group Mean 


Control 
Group Mean 


Difference 

(Estimated 

Impact) 


Effect Size 


/7-value 


Parents 


2.06 


3.00 


94 ** 


-.27 


.00 


Students 


1.90 


1.93 


-.02 


-.01 


.87 



** Statistically significant at the 99 percent confidence level. 

NOTES: Means are regression-adjusted using a consistent set of baseline covariates. Effect sizes are displayed in tenns of 

standard deviations of the study control group distribution. Valid N for parent survey = 1,555. Valid N for student 
survey = 1,025. Parent and student survey weights were used. Survey given to students in grades 4-12. 



Table 5. Year 2 Impact Estimates of the Offer of a Scholarship on the Full Sample: Parent and 
Student Reports of Satisfaction with Their School (ITT) 



Outcome 


Treatment 
Group Mean 


Control Group 
Mean 


Difference 

(Estimated 

Impact) 


Effect Size 


/7-value 


Parents who gave school a 
grade of A or B 


.76 


.63 


.13** 


.26 


.00 


Average grade parent gave 
school (5.0 scale) 


4.02 


3.73 


29 ** 


.29 


.00 


School satisfaction scale 


26.12 


23.44 


2.67** 


.33 


.00 


Students who gave school a 
grade of A or B 


.71 


.68 


.03 


.05 


.49 


Average grade student gave 
school (5.0 scale) 


3.97 


3.84 


.13 


.12 


.14 


School satisfaction scale 


34.12 


33.24 


.88 


.13 


.10 



** Statistically significant at the 99 percent confidence level. 

NOTES: Means are regression-adjusted using a consistent set of baseline covariates. Effect sizes are displayed in terms of 

standard deviations of the study control group distribution. Valid N for parent measure of school grade = 1,549; 
parent satisfaction = 1,571. Parent survey weights were used. Parent school satisfaction scale was IRT scored and had 
a range of .96 to 35.43. Valid N for student measure of school grade = 974; student satisfaction = 1,042. Student 
survey weights were used. School satisfaction scale was IRT scored and had a range of 9.67 to 46.89. Impact 
estimates reported for the dichotomous variable “parents who gave school a grade of A or B” are reported as marginal 
effects. 
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Impacts on Subgroups 



In addition to determining the general impacts of the OSP on all study participants, this 
evaluation also reports Programmatic impacts on policy-relevant subgroups of students. The subgroups 
were designated prior to data collection and include students who were attending SIN1 versus non-SINI 
schools at application, those relatively higher or lower performing at baseline, girls or boys, elementary 
versus high school students, and those from application cohort 1 or cohort 2. Since the subgroup analysis 
involves significance tests across multiple comparisons of treatment and control students, some of which 
may be statistically significant merely by chance, these subgroup-specific results should be interpreted 
with caution. Specifically: 



Subgroup Achievement Impacts 

• There were no statistically significant reading (ES = -.00) or math (ES = .05) 
achievement impacts for the high-priority subgroup of students who had attended a 
SINI public school under No Child Left Behind ( NCLB ) before applying to the 
Program. 

• The Program may have had a positive impact on reading test scores in year 2 for three 
subgroups of students, although the statistical significance of the findings was not 
robust to adjustments for multiple comparisons: 

o Students who attended non-SINI public schools prior to application to the 
Program (56 percent of the impact sample) scored an average of 5.7 scale score 
points higher in reading (ES = .15) if they were offered the scholarship 
compared to not being offered a scholarship and 6.9 scale score points higher 
(ES = .18) if they used their scholarship compared to not being offered a 
scholarship. 

o Students who entered the Program in the higher two-thirds of the test-score 
performance distribution at baseline (66 percent of the impact sample) scored an 
average of 5.2 scale score points higher in reading (ES = .15) if they were 
offered a scholarship compared to not being offered a scholarship and 6.3 scale 
score points higher (ES = .18) if they used their scholarship compared to not 
being offered a scholarship. 

o Students from the first cohort of applicants (21 percent of the impact sample) 
scored an average of 8.7 scale score points higher in reading (ES = .27) if they 
were offered a scholarship compared to not being offered a scholarship and 12.2 
scale score points higher (ES = .37) if they used their scholarship compared to 
not being offered a scholarship. 

• The OSP had no statistically significant achievement impacts for other subgroups of 
participating students, including those in the lower third of the test-score performance 
distribution at baseline, boys, girls, elementary students, secondary students, and 
students from the second cohort of applicants (effect sizes ranging from -. 14 to . 1 1). 
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Subgroup Safety and Satisfaction Impacts 



• Eight of the 10 subgroups analyzed, including parents of the high-priority subgroup of 
students who had attended SINI schools, reported viewing their child’s school as less 
dangerous if the child was offered or using an OSP scholarship compared to not being 
offered a scholarship. Effect sizes for the impact of an offer of a scholarship on parent 
perceptions of school danger for the eight affected subgroups ranged from -.21 to -.35. 
Adjustments for multiple comparisons indicate that these eight subgroup impacts on 
parental perceptions of safety are not likely to be false discoveries. The parents of 
students who were relatively lower performing at baseline and those in high school 
were the exceptions, as they did not report lower or different levels of perceived school 
danger as a result of the treatment. 

• Consistent with the finding for students overall, none of the subgroups of students 
reported experiencing differences in dangerous activities at school if they were in the 
Program. Thus, there was no impact on students’ perceptions of school safety from 
either the offer or the use of a scholarship for any of the subgroups (effect sizes range 
from -.1 1 to .09). 

• In addition to an overall impact on parental satisfaction with their child’s school, the 
Program produced satisfaction impacts on 8 of the 10 subgroups analyzed, including 
the high-priority subgroup of parents of students who had attended SINI schools. 
Effect sizes for the impact of an offer of a scholarship on the likelihood of a parent 
grading their child’s school “A” or “B” for the eight affected subgroups ranged from 
.18 to .34. Adjustments for multiple comparisons indicate that one of these eight 
subgroup impacts (for the parents of students who were relatively lower performing at 
baseline) may have been a false discovery. The statistical significance of the other 
seven subgroup impacts on parent satisfaction with their child’s school was not 
affected by adjustments for multiple comparisons. The parents of high school students 
and those in the first cohort of applicants generally did not report higher levels of 
school satisfaction that were statistically significant as a result of the treatment (effect 
sizes range from .02 to .18). 

• With one exception, there was no impact on school satisfaction if students were offered 
a scholarship, across subgroups. The high-priority subgroup of students who applied 
from a SINI school were more likely to give their school a grade of A or B (ES = .24) 
if they were offered a scholarship compared to not being offered a scholarship, 
although adjustments for multiple comparisons indicate that this finding may be a false 
discovery. 



The Impact of the Program on Intermediate Outcomes 



Understanding the mechanisms through which the OSP does or does not affect student 
outcomes requires examining the expectations, experiences, and educational environments made possible 
by Program participation. The analysis here estimates the impact of the Program on a set of “intermediate 
outcomes” that are influenced by parents’ choice of whether to use an OSP scholarship and where to use 
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it, but are not end outcomes themselves. The method used to estimate the impacts on intermediate 
outcomes is identical to that used to estimate impacts on the key Program outcomes, such as academic 
achievement. 



Prior to data analysis, possible intermediate outcomes of the OSP were selected based on 
existing research and theory regarding scholarship programs and educational achievement. Because 24 
intermediate outcome candidates were identified through this process, the variables were organized into 
four conceptual groups or clusters to aid in the analysis. 8 

There is no way to rigorously evaluate the linkages between the intermediate outcomes and 
achievement — students are not randomly assigned to the experience of various educational conditions and 
programs. That is why any findings from this element of the study do not suggest that we have learned 
what specific factors “caused” any observed test score impacts, only that certain factors emerge from the 
analysis as possible candidates for mediating influence. The analyses are exploratory, and, given the 
number of factors analyzed, some of the statistically significant findings may be “false discoveries” (due 
to chance). 



Overall, 2 years after applying for a scholarship, the Program had an impact on 1 0 of the 24 
intermediate outcomes, 8 of which remained statistically significant after adjustments for multiple 
comparisons: 

• Home Educational Supports. The results suggest that the Program may have had an 
impact on two of four intermediate outcomes in this group. The Program appeared to 
produce a positive impact on parents’ aspirations for how far in school their child 
would go (ES = .12); however, this result may be a false discovery. The Program led to 
students’ experiencing more time spent commuting to school from their homes 
(ES = .25), a result that did not lose statistical significance after adjustments for 
multiple comparisons. There were no statistically significant differences between the 
treatment and control groups on the involvement in school reported by parents in year 
2 (ES = -06) or on the use of a tutor outside of school (ES = -07). 

• Student Motivation and Engagement. The Program had no statistically significant 
impacts on any of the six elements of this group of intermediate outcomes. Two years 
after they applied to the OSP, the treatment and control group students reported similar 



Intermediate Outcome Conceptual Grouping 1, Home Educational Supports, includes parent involvement, parent aspirations, 
out-of-school tutor usage, and school transit time. Intermediate Outcome Conceptual Grouping 2, Student Motivation and 
Engagement, includes student aspirations, attendance, tardiness, reading for fun, engagement in extracurricular activities, and 
frequency of homework. Intermediate Outcome Conceptual Grouping 3, Instructional Characteristics, includes student/teacher 
ratio, teacher attitude, challenge of classes, ability grouping, availability of tutors, in-school tutor usage, programs to assist 
students with learning disabilities or English language learners, programs for advanced learners, before-/after-school care 
programs, and enrichment programs. Intermediate Outcome Conceptual Grouping 4, School Environment, includes parent/ 
school communication, school size, percent non-white, and peer classroom behavior. 
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aspirations for future schooling (ES = -.11), frequency of doing homework (ES = -.10), 
time spent reading for fun (ES = .02), and engagement in extracurricular activities 
(ES = .08). There were no statistically significant differences in student attendance 
(ES = -. 1 1) or tardiness rates (ES = -. 1 1), as reported by parents. 

• Instructional Characteristics. The offer of a scholarship appears to have had a 
statistically significant impact on 5 of the 10 intermediate outcomes in this group. 
Being offered a scholarship led to students’ experiencing smaller classes, as measured 
by student/teacher ratios (ES = -.29). The Program also led to students’ experiencing a 
lower likelihood that their school offered either tutoring (ES = -.32) or special 
programs for children who were English language learners or had learning problems 
(ES = -.66). At the same time, however, the Program had a positive impact on the use 
of an in-school tutor, presumably in schools that made them available (ES = .13). The 
OSP also led to students’ experiencing a higher likelihood of being in a school that 
offered enrichment programs (ES = .19). The statistical significance of these five 
results was not affected by adjustments for multiple comparisons. There were no 
differences between the treatment and control groups in how students rated their 
teacher’s attitude (ES = .02) or the challenge of their classes (ES = -.04), the school’s 
use of ability grouping (ES = .13), the availability of programs for advanced learners 
(ES = . 12), or before- and after-school programs (ES = .04). 

• School Environment. The Program may have affected three of the four measures of 
school environment. Students in the treatment group experienced schools that were 
smaller (ES = -.43) and had a smaller percentage of non-white students (ES = -.39) 
than the schools of the control group, findings that were not affected by adjustments 
for multiple comparisons. Treatment group students also reported having better 
behaved peers in the classroom than did control group students (ES = .16), although 
adjustments for multiple comparisons suggest that this finding may be a false 
discovery. There were no differences in parents’ reports of how their child’s school 
communicates with them (ES = .01). 

It is important to note that the findings regarding the impacts of the OSP reflect the 
particular Program elements that evolved from the law passed by Congress, and the characteristics of 
students, families, and schools — public and private — that exist in the Nation’s capital. The same program 
implemented in another city could yield different results, and a different scholarship program in 
Washington, DC, might also produce different outcomes. 



xviii 




